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ABSTRACT 

The gap between evaluation theory and practice can be 
closed through a concrete system for effectively evaluating a 
training program to ensure that it contributes to an organization's 
success. The Training Effectiveness Evaluation (TEE) system can be 
applied to any training program in industry. It consists of three 
major elements: (1) an effectiveness evaluation plan, (2) tools for 
measuring training effectiveness, and (3) the evaluation report. The 
completed plan specifies the tools that will be used to assess 
whether the training has produced the desired results. The three 
categories of evaluation tools — satisfaction, learning, and 
performance — can be presented as three scores, one for each category. 
The completed report is a powerful tool for communicating the results 
of a training program. It provides the management decision maker with 
the necessary information for understanding the impact of a training 
program. Content includes the employee/organization performance need, 
the employee/organization performance goal, the approved solution 
with both training and nontraining component, narrative summarizing 
training effectiveness, an evaluation summary with visual 
presentation and/or comparison to performance goal, and an 
improvement proposal. (Examples and samples are appended.) (VLB) 
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Abstract 



This article closos the gap between evaluation theory and practice by 

suggesting a concrete system fcr effectively evaluating a training prograa 

so that training managers can ensure that their programs do contribute to 

« 

their organization's success. The TEE consists of three major elements: 
(1) an effectiveness evaluation plan ^ (2) tools for measuring training 
ef f ecti venessf and (3) the evaluation report . 



* Accepted for publication in the Journal of European Industrial Training . 
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Training Effectiveness Evaluation 

The gap betneen evaluation theory and practice is a serious problem 
for training in industry and business <Swan3on , 1982) . A recent 
literature revien of the sufflmative evaluation on training noted "the. 
general consensus of the authors is that nost ftuoiaiative evaluation of 
training and developnent programs is not conducted effectively at the 
present tine. It may be concluded that more attention needs to be given 
to the components of sumsative evaluation used as a basis for this review 
and to upgrade the evaluation competencies of training specialists" 
(Parkeri 1986, p. 51). 

A parad^tx facing most practicing managers of training is that their 
nontraining bosses typically neither ask for nor require formal 
evaluations. And, when these managers do evaluate, it is usually in 
response to a crisis and invariably it comes too late. A typical 
workplace scenario consists of the busy training practitioner doing what 
the company wants, feeling successful, and not being regularly required to 
prove the added value that results from training. Mith a full agenda of 
important training development and delivery tasks, the busy trainer finds 
it difficult to evaluate training. However, most important organizational 
functions regularly evaluate their progress and bottom-^line contributions 
to the enterprise. In addition, it has been clearly established that 
training effectiveness evaluation data, particularly bottom-line 
performance results, are ^he key to gaining support for the training 
function from uontraining manag^^rs (Kusy, 1986). It is clearly irrational 
to not evaluate training effectiveness. 
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The purpose of this study Mas to develop and pilot test a practical 
Training Effectiveness Evaluation (TEE) system that could be applied to 
any training program in industry. Training personnel from Control Data 
Corporation and researchers from the University of Flinnetota Training and 
Development Research Center worked together to achieve this purpose. 

Both Parker's (1986) revien of literature and Kusy's (1986) study of 
management support of training evaluation ^established the need for this 
study. In addition, the TEE is the heart of the control phase of the 
comprehensive Training Technology System developed hy Swanson It Sisson 
(1985). The other four phases of the Training Technology System are 
analyze, design, develop, and implement. 

The TEE consists of three major elements: (1) an effectiveness 
evaluation plan , (2) tools for measuring training effectiveness, and (3) \ 
the evaluation report. 
Evaluation Plan 

In TEE, planning decisions are made about ithich tools Hill be used to 
asserts whether the training program produced the desired results. The 
Effectiveness Planning Sheet presents both evaluation tools and 
effectiveness questions. The four questions represent levels of training 
effectiveness and should be asked of every training program. They are: 

1. Has the training delivered professionally? 

2. Here the learning objectives met? 

3. Was the original training need met? 

4. Mas the training valuable? 
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Thtf evaluation tools, labeled A-F on the effectiveness planning sheet, are 
used in gathering the infornation needed to answer the evaluation 
questions^ These tools neasure the satisfaction, learning, and 
performance that result froiD training and, in the case of Figure 1, focus 
on basic supervisory training. 



FIGURE 1 ABOUT HERE 



An "x' in a cell on the planning shttet indicates that this evaluation 
tool is required for all training programs. The open cells represent 
reasonable evaluation options with choices needing to be made in the 
learning and performance column's. For the knowledge area, the choice of 
using a knowledge test (2.C), an in-training performance test (2.D), or 
both, must be made. For performance, the choices focus on either cost- 
benefit analysis or performance comparisons. The completed plan requires 
a minimum of four evaluation tools: Two for satisfaction, one for 
learning, and one for performance. The tools that are selected also 
address the four effectiveness questions. 

The completed QlarL specifies the tools that will be used to assess 
whether the training has produced the desired results. Figure 1 is a nlan 
for a basic supervisory training course. The sample plan sho»«s that the 
effectiveness of this course will be evaluated using the following 
measures! trainee satisfaction, trainee supervisor satisfaction, knowledge 
test, performance comparisons, and cost-benefit analysis. 
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Tools for HLasurinq Training Effectiveness 

The three categories of evaluation tooln — satisfaction, learning, and 
perfornance — can be presented as three scores, one. for each category. The 
satisfaction score is an indicator of hoK pleased trainees and their 
supervisors Here with the training; the learning snore is an indicator of 
the amount of knowledge acquired by the trainees during the training 
course; and the performance score is an indicator of the effects that 
result froffi the training. Although there are eany options available to 
prof €^ssional trainers for constructing evaluation tools, the TEE focuses 
on a limited number of reasonable options, not every option. 

The TEE requires that trainee satisfaction be measured for every 
training course. Trainee satisfaction is measured by having each trainee 
complete the Training Program Evaluation Form (Figure 2). The trainee 
satisfaction score is calculated by tallying all the trainees' responses 
to questions 1. through 7. Ordinal values are then assign^^^d to the 
following descriptors: Very good (4), good (3) , fair (2) « and poor (1). 
The overall trainee satisfaction score is obtained by averaging the scores 
and determining the mean satisfaction score which will fall within the l*-4 
range. Sub-scores on the individual questions can also be computed this 
way. 



FIGURE 2 ABOUT HERE 



The comments written by the trainee^ on the trainee satisfaction form 



Training Effecti vtness Evaluation 

7 

are not included in the trainee satisfaction score, but instead provide 
ianediate, open-ended feedback for the instructor. 

Trainee supervisor satisfaction is . measured by using the Hanagenent - 
Evaluation Fora (Figure 3). This fora is coapleted by each trainee's 
supervisor. After* the responses are gathered^ the average supervisor 
satisfaction score for the training program is computed in the saatf manner 
that average trainee satisfaction score and sub-scores are dttersined. 
Again, the Hritten comments provide the trainer with ioaediatf^, open-ended 
feedback. 



FIGURE 3 ABOUT HERE 



The tot^A satisfaction score for the training is computed by 
averaging the trainee satisfaction score and the trainee supervisor score 
and dividing this number by 2. This process weights the opinions of both 
trainees and the supervisors equally. The trainer can report the ran 
satisfaction score on the 4-point scale or use basic mathematic formulas 
to express the fcore as a ratio or percentage.^ 

Using standard trainee and supervisor satisfaction forms for ^all 
training courses allows for the comparisons of training courses with each 
other and across time, making it easy to identify and document recurring 
problems and/or successes. 

Learning in training is measured by knoNletiige tests, performance 
tests, or both. Knowledge tests measure the cognitive information learned 
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by trainees Tho types of knoHledge t«st items — multiple choice and 
matching— are encouraged because they can be scored objectively and are 
not as suseptable to guessing. In constructing knoHledge tests, care nust 
be taken to ensure that the tests produce valid and reliable results. A 
test is v^j;d Hhen it measures Hhat it is supposed to neisure and it is 
reliable Hhen it produces consistent results. The job aid for 
constructing knoHledge tests (Figure 4) includes saople test itemsi 
validity and reliability criteria, and helpful test construction 
references. 



FIGURE 4 ABOUT HERE 



In-training performance tests neasure Hhat the trainees can do by 
exaeining either the products that the trainees produce or the processes 
used by the trainees to exhibit learning. An in-training performance test 
must also be valid and reliable. The job aid for constructing in- 
training perfornance tests (Figure 5) provides examples, criteria for 
validity and reliability, and helpful performance test references. 



FIGURE 3 ABOUT HERE 



9 



Training E-Fvecti veness Evaluation 

The scores obtained Mith the knoMledge test^ the in-training 
performance test, or both, are used in calculating the total learning, 
score. When learning infor«)ation is collected using a single tool, that 
score becones the total learning score. When both tools are used, the 
learning score is calculated by computing the percentage correct score for 
each test and then adding these scores together and dividing by 2. The 
trainer can either report the raM learning score or express it as a ratio 
or percentage (see Footnote 1). 

In TEE, the tools for aeasuring the performance that results fron 
training are performance comparisons and cost-benefit analysis. 
Performance comparisons contrast the productivity of either the 
organization or the employee after training Mith the productivity before 
training or against a goal. Figure i is the job aid for performance 
comparisons. 



FIGURE 6 ABOUT HERE 



Cost-benefit analysis is used to determine the economic value of the 
training program. The benefit of a training program is determined by 
subtracting the cost of the training program from the performance value 
resulting from the program. Figure 7 is the TEE job aid for conducting 
cost-benefit analysis of training programs. 
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FIGURE 7 ABOUT HERE 



In situations Hhere 'there is ongt Measure of perforttance, that Aea^ure 
becomes the perf ornatice scjre. Jn situations Hhere both aeasures of 
perfor«ance are used and a composite score is needed, the total scores for 
both measures can be translated into like terDs, or standard scores, added 
together, and divided by two* Perforaance scores can be expressed in a 
variety of Nays including ratios, percentages, dollars, and units 
produced. It is iiaportant to express perforaance aeasures in terAS that 
have meaning to the organization. 
Effectiveness Evaluation Report 

TEE also includes systenatic reporting of training prograa 
evaluations. The report contains the categories of information that 
training and nontraining Aaoagers Aust knon in order to nake sound 
decisions. The content of the re nort includes the original 
enploy&e/organization performance need (described in 25 to 75 words), the 
employee/organization performance goal (summarized in 25 to 50 Nords) , the 
approved solution Hith both training and nontraining components (described 
in 25 to 50 Hords) , narrative summarizing the effectiveness of the 
training (25 to 50 Nord defscriptions each for the measures of 
satisfaction, learning, and performance), an evaluation summary nith 
visual presentation and/or comparison to performance goal, and an 
improvement proposal. 
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The coKpleted effectiveness Evaluation Report, as illustrated in t^e 
circuit troubleshooting training saiiple (Figure 8), is a powerful tool for 
coAAunicating the results of a training program. It provides the 
Ounav^eoent decision Maker with the necessary irifornation for understanding 
the inpact of a training prograa. > 



The TEE provides tools for planning evaluations, gathering tho 
effectiveness information, and reporting the information. Through 
systematic analysis-and reporting of effectiveness evaluations, training 
managers can ensure that their programs contribute to their organization's 
bottom line. 



FIGURE S ABOUT HERE 



Summary 
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Training Ef-f ecti venc9ss Evaluation 
Footnotes 

^It should be noted, however, that a cosposite score is questionable 
unless the individual test scores that coaprisc^ it coas fron tests xith 
similar score units, standard deviations and levels of difficulty for test 
items. The coiposit« score for tMO di'ssinilar tests is coiputed by 
determining the z-*scorfi for each test, combining th& scores, and dividing 
by 2. Additional discussion of z-scores, including the methods for 
computing them, can be found in Fundamental Research Statistics for the 
Behavioral Sciences (Roscoe, 1975). 
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Training E-f fecti vensss Evaluation 
Figure Captions 

Figure 1, Basic supervisory training program effectiveness plan. 

Figure 2* Instrument for trainee to evaluate training. 

Figure 3« Instruiient for trainee supervisor to evaluate training. 

Figure 4. Knowledge test job aid. 

Figure 5. Performance test job aid. 



Figure 6. Performance comparison job aid. 



Figure 7. Cost-benefit analysis job aid. 



Figure 8. A sample effectiveness evaluation report. 
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"Bajlc Supervisory Training Program Effectiveness Plan" 



This planning sheet helps to specify the evaluation tools that will be used to answer the 
four quesQons about the training effectiveness of each ti aining program The questions 
represent four levels of training effectiveness. The x's in tiie plaL^shee cel?s inS^^^ 

r^ Lon^Swff f ^^^"^ ^"^S programs. V o^ n cells "present 

reasonab e effectiveness evaluation options with two choices needinrto be made In 

n^Tf '5°'^^ °^ "1"^ ' knowledge test (2.C). a performance test"(2.D). 
^(iSr 4 F ' P^^°™a"ce. the choice is within cells 3.E. 3.F. 4.E. 

ii^^^i5™^*f^^^-^^°"^?, be one evaluation tool each for satisfaction, learning, and 
performance. Additionally, the selected tool, must minimally address thTfolfr^estions. 



EVALUATION TOOLS 



PROGRAM TITLE Basic Sunervisorv Tramiyp 



PREPARED BY 
APPROVED BY 



John Mart 



Sara Jameson 



DATE 



3/27/86 



EFFECTIVENESS QUESTIONS 



L DELIVERY. Was the training 

delivered professionally? 




2. objectives; Were the learning 
_^ obiectives met? 



3. NEED. Was the original need, met? 



4. VALUE. Was the training valu^bl^? 



X = required of all training programs 
O = choices for this program 



REQUIRED CHOICE #1 CHOICE #2 
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TRAINING PROGRAM EVALUATION FORM 
Training Technology System 

PROGRAM TirLE 



INSTRUCTOR(S) 



DATE 



"Please answer the following questions to help us improve future training programs," 

Veiy 

Good Good Fair Poor 

1. Quality of instructor's presentations 

2. Quality of the informatior presented _ 

3. Amount of time to practice new 

material 

4. Quality of feedback on your 

performance during training 

5. Quality of training environment ^ 

6. Usefuliiess of the course content 

toyourjoS . 

7. Was attending this training 

program a good use of your time? 

What was the most valuable part of this comxe for you? 



What was the least valuable part of this course for you? 



If you rated any item "poor", please provide some additional explanation. 



Additional comments would be appreciated 



Trainee Name (optional) 

J9 



MANAGEMENT EVALUATION OF TRAINING 
Training Technology System 

PROGRAM TrrLE:„^ 

DATES OF TRAINING: 

TRAINEE(S): 

TRAINEE SUPERVISOR: DATE: 

"Please answer the foUowng questions to help us improve future training programs." 

Now that your employee has completed training and is back on the job, what is your impression of 
the effectiveness of the training program? 

Strongly Strongly 

Agree Agree Disagree Disagree 

1. Employees have performed better 
at their old job or have been able 

perform a new job following training. ^ 

2. Attending the training was> a good 

use of the employee's tim.e. 



Additional comments would be appreciated. 
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KNOWLEDGE TEST 

TYPES OF ITEMS: 

1 . Multiple choice (samples): 

To speed up nut turning on tasks where space is limited or 
where bolts with long threads prevent the use of sockets 
"sethe, wrench. 

A. crescent 

B. combination 

C. ratchet 

D. aUen 



2. Matching (samples): 

For each item, write a number to indicate that 
the statement rpplles to: 

1. Norm-referenced assessment 

2. Criterion-referenced assessment 

3. Both norm and criterion-referenced assessment 

4. Neither norm nor criterion-referenced assessment 

Assessment in mastery-bp.5ed. 

P«>ple must fail; otherwise assessment is loo easy 

^ Assessment is useful for making predictions. 



Indicate th; best answer by circling the number. 
The most important property of an objective 
test is: 

1. Ease of marking. 

2. Accuracy of scoring. 

3. Its reliability. 

4. Its validity, 

5. Complete sampling of the syllabus. 



Directions: 



Column A contains a list of advanUges of 
varied shopping outlets. Choose from 
Column B the ouUet which best fits each 
advanuge in column A and insert the 
identifying letter in the space provided. 
Responses in column B may be used 
more than once. 



Column A 

1) "One stop" shopping 

2) Offers 24-hour service 

3) Armchair shopping 

4) All prices may be lower 

5) Product demonstrated at home 

6) Open counter dis^ilay 



CONTENT VALIDITY: 

(...does the test ineasure what it is supposed to measure?) 



Column B 

a) Mail order 

b) Door-to-door 

c) Vending machine 

d) Department store 

e) Specialty 

f) Used clothing 




Unit#l 



Unit #2 



Unit #3 



Unit #4 



TOTALS 

(# and % of test items) 



Low 



20 50% 



High 



20 50% 



TOTALS 
(# and % of test items) 



J2 3fl2L 



-15% 



-14. 



20% 



40 100% 



RELIABILITY: 

(... does the test yield consistent results'>) 
1. Use at least 25 test items for any one test 

un?eSo3e ^"^ ^'^^ '^^^ ^^^^ 

REFERENCES: * ' 
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IN-TRAINING PERFORMANCE TEST 
TYPES: 

1 . Process Checksheet (samples): 

COURSE: Buic T«u ud Htrtw« 
rc»ro»MANCE CHECKLIST 
TERMINALPEKPOIlMANCEOBJECnVE: 





Slept 


1 


factory 


Criicrlc 




i Locac iba adjuumeu koSb"^ 




0 
0 


11 »• b«t eoi of tofnue jcrew.^ 
drivrr hiKlle 




3. Loc4i« torque iindictior lifht 




0 


wo»P«iwn puiuc colUr u mid- 
poiot of toiqac lotwdriver 




y Uat up loiQHt ijHjicator lii^T ' 


1 


0 
0 


cloctwije/cwiweretockwije to 30 
iaclypotudi 

« U» 31 toctVpOUDd lettint — 




6. Lock the aajtuunrat kiwK 

laitn Hiuiipi bead tip {mo ' 

poiition 


1 
1 


o o 


■djmmmn fob tockcrt in piaM 
M »• <od of toi^»e icrcwSriver 
oppofite adjuitnirm knnk 




1. t^uitioo autw 


1 


0 


ioto fl')preanlk4l holeia 




y. rosiuosriMwuhcr 
i6. foiiuoa btx DM 


1 


0 


mbmembly fwrtpieo 

™ HUM or pnpouuSS^^St^ 




n.Fiitea Mrtwift in 

12. PoiiUoo torqut icrtthJrjvtr " 

13. Tigtuco icnw ■ 

_: 


1 
1 


0 

0 
0 

0 
0 


om sfuok of pRposiuoQcd icrtw" 
iaitepg 

finiit (iihicn 

Mital io acnw head 
«nUI cl«kiB| iOMtid occuw 



Product Specifications (samples): 




MAhf AGEMENT DELEGATING 
RESPONSIBILrrY 



1. Explaint the oew retponjibility 
to the ecjployee tod telh why it is 
iffiportaoL 

Z TellJ the employee the pcrfonnince 
jUDdtrtJ that are expected for the tasks 

3. Aiki the enyloyce if there are 
«ny questioni or juggcsUoos and res- 
poods to the coacemi that are exprejvtd. 

4. Asu the employee to make a com- 
mittmeiit to the respouibility. 

5. Jells the employee that you have 
coofideace m their abOiiy to cany out 
the respoDiibility. 



PERF. 

RATING 

GCX)D-3 

0.K..2 

POOR.l 

NO.0 



Total 



DirecUons: Rate the mstrumcnt according to the following 
cntena by placmg a-?r in the appropriai blaSk. ^ 

CRrreRION DESCRIPTION YES NO 

QwUQr Dxi it measure quaU^ of 

the performance? 

SkiU 

Attitude 

Does U measure efficiency 

of the operaUon? 

Docs the lanjuaje, design, ' 

length, and degree of detail 

proEDoteeascofuse? 

Dxs it achieve the goals of *" 

monitoring ttudeot programs, 
di^oiing, ccrtiiyiiig,and 
cvtluAtiQ^ iotirttctioaa? 

DociiiiciYe,withUak * " 

rcvisioo, for self-evaluation, 
peer evaluation, and instructor/ 
Supervisor evaluation? . 

Does tt measjire what it was ^ 

designed sa measure? 

Does it provide tnutworthy 
or consistent measure? 
Recommeodations for Change 



Efficiency 
Ease of use 



Achievement 
uf Goals 



Adaptability 

Validity 
Reliabili^ 



CONTENT VAUDITY- 

1 Make ^^'^ I 

L J^S'beSrl^^sr^ ''^'^''^^ ^ <^^^^ steps specified by 

I thel&tion^^teria'^'^ specifications, quality and quantity, axe included 

RELIABILrrY: 

(...does the test yield consistent results?) 

describe the siwdficaK te^S.^,Sf *'f"~«^^*Wle doing {."or 
REFERENCES: 

Wentling. T. LVrnduS^u I h ?19"jM'"f!Ji:3i;'"T'^''"'''lll'm- ^^W. 28-37. 
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BEST COPY AVAILABLE 



PERFORMANCE COMPARISION 
TYPES: 

1. Employee Job Performance (samples): 

... (same information as presented on the Performance Test) 

Organization Performance (samples): 



2. 



1 



The Crtlnin J suff decided to evtlutie the cffccUvewii of ihcncedi 
discovery iriiDing progrtm by considering wheihcr training made 
an impact on 1985 aaJea. To conduct ihU evaluition. they utUized 
a design whkh examined tales voluoe per month acn>is each sales 

n?.m>t^f I'JtlS^^ ^" eonUogent on the Stic, 

District of which the sales teptesenuUve wis a meraber The stag, 
gered line «Pj«cni» the actual training program which occurred over 
a two day period. The foUowing Hgurt iUusiraies thiy method. 

Figure. Sales per tnooth before and after training * 



Mid»c*t 
DUuictA 



Midwetl 
Stiri 

DiiirictB 



Midwest 
Stlci 



5.0, 
4.3 
4.0 
3.3 

3.0 ; 

2.3. 
2.0 
1.3 
1.0. 
.5. 



3.0. 

4.0. 
3J. 
3.0. 
2.5. 
2.0 
IJ. 
1.0 
-5. 



3.0 
4.3 
4.0 
3-5-1 
3.0 
2J. 
2.0. 
1.5. 
1.0 
J. 



Before 
Training 



Before 
Training 



After 
Training 



Before 
Training 



AHct 
Training 



After 
Training 



iT I I — I — I — I — r*n — I — r 

JFMA MJJASOM 
Tune (io months) for 1983 



COMPARATIVE MANUFACTURING PRODUCnYTTY 



Superior A 


Superiors 


Superior C 


Employee 


Hrly. 


Employee 


Hrly. 


Employee 


Hrly. 


DO. 


piod. 


no. 


prod. 


no. 


pfod. 


1 


163 


11 


194 


21 


172 


2 


149 


12 


138 


22 


137 


3 


118 


13 


137 


23 


136 


4 


108 


14 


131 


24 


135 


5 


106 


15 


110 


25 


127 


6 


93 


16 


83 


26 


100 


7 


60 


17 


f.l 


27 


56 


8 


57 


18 


49 


28 


52 


9 


42 


19 


48 


29 


41 


10 


30 


20 


41 


30 


28 


Average 


92,6 


Average 


99.8 


Average 


98.4 



CONTENT VALIDITY: 

(...does the test measure what it is supposed to measure?) 
1. Detmnine if the organization regularly collects data on the performance of the 
work group in the area under investigation. 

nf '"^ ""ii °^ performance selected is the same or a good approximation 

of the performance need specified in the original needs assessment. 

RELIABILITY: 

(...does the test yield consistent results?) 

methSfi^"'® organizational records, inquire about the reUability of the data collection 

as comparision of group performance during earlier time 
periods before and after the program. 

REFERENCES: 

Gilbert, T. F. (1978). Human Cnnipptfnfff, New York: McGraw-Hill. 
Sa^^J!: gffWtS 9f types of training evaluation on snp pnrt nf 

t raining among gWIPgra g mannffn Unpublished doctoral SiesisTunffity 
of Minnesota, Minneapolis. .wmyciauy 
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COST.BENEFIT ANALYSIS 
TYPES 

1- Cost Analysis 

COST analysts; ^npi; .if^ppj 

Forecaster 

1. Nwds Analysis/plannine 
Staff 

External consultant costs 
Materials 



Date 



2. Work behavior analysis 
Staff 

External consultant costs 
Materials 

3. Design 

Staff 

External consultant costs 
Materials 

External support costs 

4, Development 

Staff 

External consultant costs 
Materials 



5. Implementation 
Trainee 
Facilities 
Tuition/fees 
Staff 
Materials 



Evaluation 
Staff 

External consultant costs 



Subtotal 



Subtotal $" 



Subtotal 



Subtotal 



Subtotal $" 



Subtotal 
Total 




2. Performance Valuing 
NET PERFORM ANCF VALUE cat ri it ajj on work 

A. Data Required for Calculations 

(a) What is the desired performance as a result of 
worker training? 

(b) What unit(8) of measure will be used to describe 
the performance? 

(c) What is the dollar value that wiU be assigned to 
each unit of measure? 

(d) What is the estimated training time to reach 
the goal? 

(e) What is the current level of worker performance? 

(f) How many workers will participate in the training? 

B. Calculations to Determine Net Performance Value 

(g) What is the estimated performance level during 
training? WUl trainee produce during training? 

a 4- e 

No*0 Yes= 2 

(h) What is the length of the period being evaluated 
(at a minimum this wUl be the longest "d" of all 
options under consideration)? 

(i) What is the estimate of the total number of units (b) 
that WiU be achieved during training? [d x g] 

0) What IS the estimate of the total individual pcr- 

fonnimcc for the evaluation period? [(h - d) x a] + i 
(k) What IS the value for the total performance for the 

evaluation period? [cxj] 
(1) What is the net performance value sain*' 

[k.(excxh)] ' 
(m) Do you want to calculate the total net performance 

value of all trainees? 

Ycs-lxf 



3. 



Cost-Benefit Model 
Performance Value 

: Cost 

~ • Benefit 



^ No K Net Performance Value of one 
trainee which is value of "1" 



7. Total costs 

(turn of all subtotals) 

COmENT VAUDITY: 

(...does Ae test measure what it is supposed to measure?) 
RELIABILITY: 

(...does the test yield consistent results?) 

1. Double check the individual numbers and their manipulation in the formula 

2. Have a second analyst prepare a cost-benefit analysis. 

REFERENCES: 

[forecasting training costs and benefits. 

'^forf^^l'^^^^f: ^^^^^^^ 

^""^fSEl.^-' * Compton, T.,(1981). Assessing costs, benefits and productivity in 

traimng systems. Training iinri TV vrfnpmi?nf Tnt^^ flj 3^(1)^ 52!?^ ^ 



EFFECTIVENESS EVALUATION REPORT 



Program Tide: Circuit Troubleshooting 

Program Date(s): 2/20/86 



Department: Technical Training DeparfmpT^t 

Pr^aredBy: MarkBaber 



Distributed To: James Birt. Mark Olser, Rnh Drft-v 



1. ORIGINAL EMPLOYEE/ORGANIZATION PERFORMANCE NEED 

The timeliness of repairs in the circuit areas was not sufficient to meet the schedule 
demands: average thru-put time was 115 hours. The first-fix repair rate was 68% and 
the additional repair process resulted in equipment being uimecessarily damaged. 



2. EMPLOYEL. ORGANIZATION PERFORMANCE GOAL ' 

Training goals were a first-fix rate of 80% and an average thru-put time of 59 hours. 
Availability of CE-4 insertion tools was expected to improve the thm-put time by 4 hours 
and the revised part orderinjg system was expected to improve thru-put time by 2 hours. 



3. APPROVED SOLUTION (TRAINING AND NON-TRAWING COMPONENTS) 

Peters approved circuit troubleshooting training for the 61 test technicians and 
indicated that CE-4 insertion tools would be available for aU trainees. She also approved 
the implementation of the revised system for part ordering. (Memo 1/86) 



4. EFFECTIVENESS OF TRAINING 

The effectiveness of the Circuit Troubleshooting Training was measured firom the 
perspectives of satisfaction, learning, and performance. 

Satisfaction measurements were obtained from the trainees and from their supervisors. 
The traLiees mean rating for delivery effectiveness was 62%, the quality of information 
presented rating was 50%, and the usefulness of this training to their jobs rating was 90%. 
The overall management rating of this course was 70.5%. 

Learning during training was measured by a knowledge test and by performance 
in-training. The comparison of the pre and post test of knowledge showed an increase 
of 57.6% for the group. The in-training performance was measured with lab tests. The 
average trainee score was 96%. 

Performance measurements showed a 15% improvement for first-fix efficiency and 
a 57 hour improvement on thru-put time. Cost-benefit analysis showed a training benefit 
of $715,365. 
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5. EVALUATION SUMMARY 
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IMPACT OF TRAINING 
ON 

imU-PUTTIME 
(Goal - 59 hours) 





BEFORE TRAINING AFTER TRAINING 
THRU-PUT TIME 



HOURS 



The data represented in the graphs was obtained from the SQC records of the test 
department The time period for the "before training" data is 30 days and the time period 
for the "after training" data is 45 days. 



6. IMPROVEMENT PROPOSAL 

L Trainees indicated that th^ wunt more feedback on their perfoimance during 
training. Instructors will respond to this suggestion. 

2/ CE-4 insertion tools are still needed for approximately 10 test technicians. 

26. 



